Worth pointing out something about query engines and dataframe libraries deciding to not do so for internal/intermediate values in computations but applying a canonicalization pass when data leaves the system.

Perhaps we can emphasize that all bytes in an Arrow array, regardless if they are "reachable", are readable by other libraries and users. Thus they should contain no potentially sensitive data (like uninitialized values).

And therefore, if query engines choose to use uninitialized memory internally as an optimization, they should ensure all such uninitialized values are cleared before passing the Arrays to another system

I suggest we also make the point about performance here to give context about why
validation is not always performed

Perhaps something like this:

"Arrow implementations often assume Arrays follow the specification
to provide high speed processing. It is extremely important that
your application either trusts or validates the Arrays it receives from
other sources.

Many Arrow implementations provide APIs to do such validation.

In terms of APIs, the Rust implementation always validates data from external sources, unless the validation is explicitly turned off with APIs marked as unsafe (a special Rust keyword).

Perhaps we can emphasize that all bytes in an Arrow array, regardless if they are "reachable", are readable by other libraries and users. Thus they should contain no potentially sensitive data (like uninitialized values).

And therefore, if query engines choose to use uninitialized memory internally as an optimization, they should ensure all such uninitialized values are cleared before passing the Arrays to another system

I don't think this is any different than the other APIs -- basically "if you don't trust the producer source, you should always explicitly validate the arrays before processing them"

This doesn't seem any different for the C Data Interface than for the other APIs (like IPC files. etc)

Hmm, that's true. I might just remove if it muddies the message.

As above, I think we could combine this into the section about validating data from untrusted sources, and give C Data Interface and IPC Format as examples of potentially untrusted sources.

"This" means the IPC format section?

Yes -- I was thinking that if the guidance is the same for IPC and C Data Interface, calling them out separately makes this documentation more verbose than it could be

The high level overview in my mind is:

1. If you read Arrow data (in any format) from an untrusted source, it can be a memory and security concern. Thus you should always verify such data.
2. Implementers should provide a way to let users opt in / out of the verification depending on the source and their security threat model.

If there are specific things to suggest remembering to check that is specific for specific formats such as the FlagBuffers offsets for IPC based formats, then we could call them out in those sections.

I am happy to propose some new wording if you like

Well, the guidance is not the same. For the C Data Interface, it is simply impossible (to my knowledge) to protect against rogue raw pointers. For IPC, validation is possible to ensure safe operation.

Yes, there's no way to meaningfully validate an ArrowArray other than perhaps to check pointers for unexpected NULL. This is because the ArrowArray does not provide buffer lengths (the consumer has to infer those based on the ArrowSchema and/or the buffer data of previous buffers). This is roughly the same as receiving an Arrow C++ or arrow-rs array from another library: the consumer has to assume it was correctly produced to avoid a crash and a malicious consumer can always attempt to read byte ranges it isn't supposed to based on the buffer pointers).

Hmm, that changes the meaning quite a bit. I wanted to stress that "users" don't directly implement the Arrow specs, they use language-specific abstraction layers provided by an implementation. Perhaps I should just use these words :)

I've tried to improve the wording a bit.

The new wording looks good to me 👍

1. users of Arrow: that is, developers of third-party libraries or applications
   that don't implement directly implement the Arrow formats or protocols, but
   instead call language-specific APIs provided by an Arrow library
   (as defined below);

It seems like links to the invalid fuzz data has been added above

I personally think this PR is already hugely valuable, even without such a list.

Thus I suggest we merge this PR and get it published. We can file an issue to track adding type specific security risks as a follow on

BTW the way the Rust API works is that by default, data read from IPC is explicitly validated (which is indeed quite expensive)

It is possible to turn this validation off via an unsafe API (the unsafe bit means the calling application has to explicitly disable the validation to acknowledge they are trusting the source):

https://docs.rs/arrow-ipc/57.2.0/arrow_ipc/reader/struct.FileDecoder.html#method.with_skip_validation

This is good and clear 👍